Skip to content

fix(gpu): tear down cuda-lts VHD prebake on GRID nodes before managed install - #8918

Closed
Ganeshkumar Ashokavardhanan (ganeshkumarashok) wants to merge 3 commits into
official/v20260702from
ganesh/grid-prebake-teardown-0702
Closed

fix(gpu): tear down cuda-lts VHD prebake on GRID nodes before managed install#8918
Ganeshkumar Ashokavardhanan (ganeshkumarashok) wants to merge 3 commits into
official/v20260702from
ganesh/grid-prebake-teardown-0702

Conversation

@ganeshkumarashok

@ganeshkumarashok Ganeshkumar Ashokavardhanan (ganeshkumarashok) commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes nvidia-smi failing with "Failed to initialize NVML: Driver/library version mismatch" on agentpool GRID / converged A10 (NVv5) nodes and NAP GRID nodes booting the CUDA-prebaked shared Ubuntu VHD.

Root cause: the shared Ubuntu VHD prebakes only the cuda(-lts) driver + a DKMS marker (#8786/#8803). The existing cleanUpPrebakedGPUDriver only runs on nodes that do not install a managed driver (GPU_NODE != true OR skip). A GRID node installs the grid driver on top of the stale prebaked cuda module + /usr/bin/lib64 libs, and the two collide → NVML mismatch.

Fix

Add cleanUpGridNodeCudaPrebake, invoked from ensureGPUDrivers before the configGPUDrivers / validateGPUDrivers dispatch (so both install paths are covered). It tears down the prebake when:

  • this node's driver type maps to grid (grid / grid-v20), and
  • the prebake marker is not grid — i.e. a cuda marker, or a legacy marker with no driver_kind= line (the only kind the VHD bakes today).

This is a pure driver-kind mismatch, so no version comparison is needed. Reuses the existing cleanUpPrebakedGPUDriver for removal (which also clears /usr/bin/lib64 and the DKMS registration).

Scope

GRID mismatch only. CUDA nodes are intentionally untouched — the NAP cuda same-kind / different-version case (580.126.09 vs baked 580.159.04) is a separate issue, addressed by aligning the driver version NAP requests with the baked image rather than by teardown.

Why this is safe on official/v20260702

Consume/skip-build is not in this branch, so a kept prebake buys no provisioning benefit today; the only effect of the guard is removing a driver that would otherwise break a GRID node. On a matching (grid) or absent marker, or on a CUDA node, it is a no-op.

Test plan

  • shellcheck clean on the new function
  • Isolated logic harness: teardown on grid+cuda-marker, grid-v20+cuda, grid+legacy-marker; no-op on cuda-lts node, grid+grid-marker, no marker, non-Ubuntu
  • Added ShellSpec coverage in cse_config_spec.sh (cleanUpGridNodeCudaPrebake)
  • e2e on an agentpool GRID/A10 node + a NAP GRID node booting the prebaked VHD

🤖 Generated with Claude Code

@ganeshkumarashok
fix(gpu): tear down cuda-lts VHD prebake on GRID nodes before managed…
f5bed7d 
… install

The shared Ubuntu VHD prebakes only the cuda(-lts) driver plus a DKMS marker.
A GRID / converged (A10, NVv5) node then installs the grid driver on top, and
the stale prebaked cuda module + its /usr/bin/lib64 userspace libs collide with
the grid driver, so nvidia-smi fails with "Failed to initialize NVML:
Driver/library version mismatch".

The existing cleanUpPrebakedGPUDriver only runs on nodes that do NOT install a
managed driver (GPU_NODE != true OR skip), so managed GRID nodes keep the stale
prebake. This adds cleanUpGridNodeCudaPrebake, run from ensureGPUDrivers before
the configGPUDrivers/validateGPUDrivers dispatch (so both paths are covered),
which tears down the prebake when this node installs grid and the marker is not
grid (a cuda or legacy kind-less marker). Pure driver-kind mismatch, so no
version comparison is needed. CUDA nodes are intentionally untouched.

Scoped to the GRID mismatch only; the NAP cuda same-kind/different-version case
is addressed separately by aligning the driver version NAP requests with the
baked image.
Copilot AI review requested due to automatic review settings July 13, 2026 21:05
Copilot AI reviewed Jul 13, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates the Linux CSE GPU-driver provisioning flow to proactively remove a mismatched CUDA(-LTS) prebaked driver on Ubuntu VHDs when the node is going to install a GRID driver, preventing NVML “driver/library version mismatch” failures on GRID/A10 (NVv5) nodes that boot shared prebaked Ubuntu images.

Changes:

  • Added cleanUpGridNodeCudaPrebake to tear down a CUDA-kind prebake marker on GRID-kind nodes before driver install/validation dispatch.
  • Invoked the new teardown step early in ensureGPUDrivers (Ubuntu-only) to cover both config and validation paths.
  • Added ShellSpec coverage for the teardown decision logic.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File Description
parts/linux/cloud-init/artifacts/cse_config.sh Adds and wires cleanUpGridNodeCudaPrebake into ensureGPUDrivers to remove mismatched CUDA prebake on GRID nodes.
spec/parts/linux/cloud-init/artifacts/cse_config_spec.sh Adds ShellSpec tests validating teardown vs no-op behavior across marker/node-kind combinations.

Copilot AI review requested due to automatic review settings July 13, 2026 21:07
Copilot AI reviewed Jul 13, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Comment thread parts/linux/cloud-init/nodecustomdata.yml
Comment on lines +39 to +43
- path: {{GetCSEConfigScriptFilepath}}
permissions: "0744"
encoding: gzip
owner: root
content: !!binary |
Merged
4 tasks
@ganeshkumarashok
test(gpu): add non-Ubuntu no-op case for cleanUpGridNodeCudaPrebake
995e244 
Addresses Copilot review on #8919: cover the OS != UBUNTU_OS_NAME early
return so a regression that removes or inverts the Ubuntu gate is caught.
Copilot AI review requested due to automatic review settings July 13, 2026 21:43
@github-actions

github-actions Bot commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Changes cached containers or packages on windows VHDs

Please get a Windows SIG member to approve.

The following dif file shows any additions or deletions from what will be cached on windows VHDs organised by VHD type.

  • Additions are new things cached.
  • Deletions are things no longer cached.
diff --git a/vhd_files/2022-containerd-gen2.txt b/vhd_files/2022-containerd-gen2.txt
index e187816..db10c9e 100644
--- a/vhd_files/2022-containerd-gen2.txt
+++ b/vhd_files/2022-containerd-gen2.txt
@@ -129,0 +130 @@ mcr.microsoft.com/oss/v2/kubernetes-csi/secrets-store/driver:v1.5.4
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.13-windows-hpc-1
@@ -131 +132 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.14-windows-hp
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.15-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.10-windows-hpc-1
@@ -133 +134 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.11-windows-hp
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.12-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.5-windows-hpc-1
@@ -135 +136 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.6-windows-hpc
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.7-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.1-windows-hpc-1
@@ -137 +137,0 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.2-windows-hpc
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.3-windows-hpc-1
diff --git a/vhd_files/2022-containerd.txt b/vhd_files/2022-containerd.txt
index 760db3c..94de353 100644
--- a/vhd_files/2022-containerd.txt
+++ b/vhd_files/2022-containerd.txt
@@ -129,0 +130 @@ mcr.microsoft.com/oss/v2/kubernetes-csi/secrets-store/driver:v1.5.4
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.13-windows-hpc-1
@@ -131 +132 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.14-windows-hp
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.15-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.10-windows-hpc-1
@@ -133 +134 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.11-windows-hp
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.12-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.5-windows-hpc-1
@@ -135 +136 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.6-windows-hpc
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.7-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.1-windows-hpc-1
@@ -137 +137,0 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.2-windows-hpc
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.3-windows-hpc-1
diff --git a/vhd_files/2025-gen2-tl.txt b/vhd_files/2025-gen2-tl.txt
index a901175..4f3c531 100644
--- a/vhd_files/2025-gen2-tl.txt
+++ b/vhd_files/2025-gen2-tl.txt
@@ -34 +33,0 @@ HKLM:\SYSTEM\CurrentControlSet\Policies\Microsoft\FeatureManagement\Overrides\18
-HKLM:\SYSTEM\CurrentControlSet\Policies\Microsoft\FeatureManagement\Overrides\908168846=1
@@ -60,0 +60 @@ mcr.microsoft.com/oss/v2/kubernetes-csi/secrets-store/driver:v1.5.4
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.13-windows-hpc-1
@@ -62 +62 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.14-windows-hp
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.15-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.10-windows-hpc-1
@@ -64 +64 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.11-windows-hp
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.12-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.5-windows-hpc-1
@@ -66 +66 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.6-windows-hpc
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.7-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.1-windows-hpc-1
@@ -68 +67,0 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.2-windows-hpc
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.3-windows-hpc-1
diff --git a/vhd_files/2025-gen2.txt b/vhd_files/2025-gen2.txt
index 4d7aa19..d0ea692 100644
--- a/vhd_files/2025-gen2.txt
+++ b/vhd_files/2025-gen2.txt
@@ -34 +33,0 @@ HKLM:\SYSTEM\CurrentControlSet\Policies\Microsoft\FeatureManagement\Overrides\18
-HKLM:\SYSTEM\CurrentControlSet\Policies\Microsoft\FeatureManagement\Overrides\908168846=1
@@ -60,0 +60 @@ mcr.microsoft.com/oss/v2/kubernetes-csi/secrets-store/driver:v1.5.4
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.13-windows-hpc-1
@@ -62 +62 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.14-windows-hp
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.15-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.10-windows-hpc-1
@@ -64 +64 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.11-windows-hp
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.12-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.5-windows-hpc-1
@@ -66 +66 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.6-windows-hpc
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.7-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.1-windows-hpc-1
@@ -68 +67,0 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.2-windows-hpc
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.3-windows-hpc-1
diff --git a/vhd_files/2025.txt b/vhd_files/2025.txt
index 1441011..ab44d8b 100644
--- a/vhd_files/2025.txt
+++ b/vhd_files/2025.txt
@@ -34 +33,0 @@ HKLM:\SYSTEM\CurrentControlSet\Policies\Microsoft\FeatureManagement\Overrides\18
-HKLM:\SYSTEM\CurrentControlSet\Policies\Microsoft\FeatureManagement\Overrides\908168846=1
@@ -60,0 +60 @@ mcr.microsoft.com/oss/v2/kubernetes-csi/secrets-store/driver:v1.5.4
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.13-windows-hpc-1
@@ -62 +62 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.14-windows-hp
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.15-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.10-windows-hpc-1
@@ -64 +64 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.11-windows-hp
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.12-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.5-windows-hpc-1
@@ -66 +66 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.6-windows-hpc
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.7-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.1-windows-hpc-1
@@ -68 +67,0 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.2-windows-hpc
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.3-windows-hpc-1

Copilot AI reviewed Jul 13, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Comment thread parts/linux/cloud-init/nodecustomdata.yml
Comment on lines +38 to +42
# ---- hotfix: auto-generated by hotfix-generate GH Action ----
- path: {{GetCSEConfigScriptFilepath}}
permissions: "0744"
encoding: gzip
owner: root
Merged
2 tasks
@ganeshkumarashok

Ganeshkumar Ashokavardhanan (ganeshkumarashok) commented Jul 13, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by #8921, which is an exact cherry-pick of the squashed main merge (#8919) onto official/v20260702 — byte-identical GRID-fix content. Closing in favor of that backport.

@aks-node-assistant

aks-node-assistant Bot commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

AgentBaker Linux gate detective

Run: https://msazure.visualstudio.com/CloudNativeCompute/_build/results?buildId=172065316
Failed job/stage/task: representative Linux VHD Test, Scan, and Cleanup jobs, exit code 2.

Detective summary: VM-side validation failed before product checks because the test VM could not fetch refs/pull/8918/merge; git fetch --quiet origin refs/pull/8918/merge:refs-pull-8918-merge returned fatal: couldn't find remote ref after retries.

Likely cause / signature: Known PR merge-ref availability race; wiki signature pr-merge-ref-race.

Confidence: High.

Recommended owner/action: Node Lifecycle pipeline infra owner should continue repair item #38759467; no PR author action unless rerun failures show product-specific validation errors.

Strongest alternative: PR source branch changed during the run; less likely as a PR code issue because the failing operation is a generic fetch of a transient GitHub PR merge ref across matrix jobs.

Evidence: ADO timeline shows many Test/Scan jobs failed with exit code 2; failed task log includes the missing remote ref and git-clone error.

Wiki signature: pr-merge-ref-race

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments
@sulixu sulixu sulixu approved these changes
@cameronmeissner Cameron Meissner (cameronmeissner) Awaiting requested review from cameronmeissner cameronmeissner is a code owner
@Devinwong Devinwong Awaiting requested review from Devinwong Devinwong is a code owner
@lilypan26 lilypan26 Awaiting requested review from lilypan26 lilypan26 is a code owner
@AbelHu Chou Hu (AbelHu) Awaiting requested review from AbelHu AbelHu is a code owner
@djsly Sylvain Boily (djsly) Awaiting requested review from djsly djsly is a code owner
@phealy Patrick W. Healy (phealy) Awaiting requested review from phealy phealy is a code owner
@r2k1 r2k1 Awaiting requested review from r2k1 r2k1 is a code owner
@timmy-wright Tim Wright (timmy-wright) Awaiting requested review from timmy-wright timmy-wright is a code owner
@zachary-bailey Zachary (zachary-bailey) Awaiting requested review from zachary-bailey zachary-bailey is a code owner
@awesomenix Nishchay (awesomenix) Awaiting requested review from awesomenix awesomenix is a code owner
@mxj220 Mark Ibrahim (mxj220) Awaiting requested review from mxj220 mxj220 is a code owner
@pdamianov-dev Peter Damianov (pdamianov-dev) Awaiting requested review from pdamianov-dev pdamianov-dev is a code owner
@calvin197 Calvin S. (calvin197) Awaiting requested review from calvin197 calvin197 is a code owner
@surajssd Suraj Deshmukh (surajssd) Awaiting requested review from surajssd
@SriHarsha001 Sri Harsha (SriHarsha001) Awaiting requested review from SriHarsha001 SriHarsha001 is a code owner
@runzhen Runzhen (runzhen) Awaiting requested review from runzhen runzhen is a code owner
@karenychen Karen Chen (karenychen) Awaiting requested review from karenychen karenychen is a code owner
@xuexu6666 Xu Xue (xuexu6666) Awaiting requested review from xuexu6666 xuexu6666 is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@ganeshkumarashok @sulixu